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Foreword 


This  effort  was  funded  by  the  Operations  Analysis  Division  (OAD)  under  program 
HQMC  AVN  APX.  The  overarching  objective  of  this  research  effort  is  to  develop  and 
validate  a  non-cognitive  profile  of  successful  US  Marine  Corps  Air  Traffic  Controllers 
(ATC)  which  will  assist  in  the  selection  and  screening  process  of  Marines  for  ATC  duty. 
This  aim  was  realized  by  a  three-phase  process,  as  discussed  in  the  previous  technical 
reports  titled  Suitability  Screening  Test  for  Marine  Corps  Air  Traffic  Controllers 
Technical  Report  and  Suitability  Screening  Test  for  Marine  Corps  Air  Traffic 
Controllers  Technical  Report  Phase  II  (NPRST-TR-13-1  and  NPRST-TR-13-2).  The 
current  report  concerns  the  implementation  of  Phase  III:  validation  of  a  noncognitive 
test  and  development  of  a  cognitive  test  prototype  to  be  used  for  a  suitability  profile  of 
success  in  the  Marine  ATC  community.  Derived  from  scores  on  the  Navy  Computer 
Adaptive  Personality  Scales  (NCAPS),  the  suitability  profile  was  used  to  predict  success 
using  Schoolhouse  and  Operational  Forces  performance  data.  Utilizing  this  type  of  non¬ 
cognitive  profile  can  greatly  increase  the  efficiency  of  the  Marine  Corps  ATC  personnel 
pipeline  and  relieve  the  burden  of  unproductive  training  through  detection  of  Marines 
who  are  and  are  not  likely  to  graduate  or  perform  satisfactorily  as  ATCs. 

The  authors  wish  to  thank  the  funding  sponsor,  and  project  officers  for  their 
assistance  in  this  project.  A  number  of  Marine  Corps  Air  Traffic  Control  instructors  at 
the  ATC-School  and  training  staff  within  the  Operational  Air  Traffic  Control  Facilities 
provided  invaluable  assistance  in  the  collection  of  academic/ qualification  data,  and  their 
accessibility  greatly  enhanced  the  development  of  the  validation  program  undertaken  in 
the  present  study. 


D.  M.  CASHBAUGH 
Director 
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Summary  Report 


Problem 

Entry  into  military  occupational  specialty  (MOS)  training  schools  requires  a 
minimum  score  requirement  on  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB),  a  test  battery  that  assesses  cognitive  abilities  through  performance  in  reading, 
mathematics,  general  science,  as  well  as  basic  knowledge  about  electronics,  mechanical 
systems,  and  automotive-shop.  The  ASVAB  was  designed  to  increase  the  effectiveness 
and  efficiency  of  the  selection  procedure,  and  determines  that  incumbents  have  the 
necessary  cognitive  skills  to  handle  the  technical  aspects  of  training  and  job 
performance.  However,  the  current  Marine  Corps  Air  Traffic  Control  selection  process 
using  the  ASVAB  has  resulted  in  cognitively  proficient  applicants,  who  are,  nevertheless, 
classified  as  “unsuited”  for  ATC  duty  during  post-training  performance.  The  inclusion  of 
individuals  into  the  ATC  MOS  who  will  ultimately  be  unfit  for  this  position  is  inefficient. 
Building  on  the  first  and  second  phase  of  the  research  effort,  which  developed  and 
validated  a  Marine  Air  Traffic  Control  Suitability  Screening  Test  (MATC-ST)  on  separate 
Operational  Forces  and  Schoolhouse  performance  metrics  (Walker,  Farmer,  &  Roberts, 
2013),  Phase  III  discusses  findings  from  validity  analysis  utilizing  integrated  Marine 
ATCs  Schoolhouse  passing  data  through  7257  qualification  data.  Phase  III  also  looked  at 
the  Federal  Aviation  Administration’s  Air  Traffic  Scenario  Test’s  (AT-ST’s)  ability  to 
predict  Air  Traffic  Control  performance.  The  ultimate  goal  is  to  create  a  more  effective 
suitability  profile,  by  combining  the  non-cognitive  algorithm  with  cognitive  test  results. 
As  such,  the  objective  is  to  increase  the  efficiency  of  the  Marine  Corps  ATC  personnel 
pipeline  and  reduce  the  burden  of  unproductive  training  by  providing  early  detection  of 
Marines  who  are  and  are  not  likely  to  perform  satisfactorily  as  ATCs. 

Objective  or  Purpose 

Individuals  possess  a  variety  of  abilities,  preferences,  interests,  and  personal 
characteristics  that  should  be  useful  in  predicting  who  will  be  best  suited  for  different 
types  of  military  jobs  or  duties.  The  overarching  objective  of  this  project  is  to  capitalize 
on  these  individual  differences  by  developing  a  non-cognitive  and  cognitive  profile  of 
successful  air  traffic  controllers  to  be  used  in  the  selection  procedures  for  Marine  Corps 
ATC  duty.  To  reach  this  objective,  test  scores  on  measures  of  non-cognitive  traits  have 
been  used  to  develop  and  validate  a  profile  (MATC-ST)  for  successful  ATC  performance 
in  the  schoolhouse  and  operating  forces.  The  current  phase  of  this  project  seeks  to 
validate  a  non-cognitive  profile  in  ATC  performance  in  the  schoolhouse  and  Operating 
Forces  along  with  developing  a  cognitive  test  for  prediction  of  successful  ATCs 
performances.  Noncognitive  validation  based  on  schoolhouse  performance  and 
Operational  Forces  7257  qualification  data  seeks  to  minimize  ATC  academically-related 
training  failures  and  Operating  Forces  revocation  to  further  emphasize  the  utility  of 
considering  non-cognitive  traits  in  conjunction  with  ASVAB  standards  when  selecting 
for  ATC  duty. 
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Approach  or  Method 


Using  a  multifaceted  adaptive  personality  measure,  the  Navy  Computer  Adaptive 
Personality  Scales  (NCAPS),  the  authors  assessed  ATCs  in  the  supporting  establishment 
operating  forces  (7257  MOS)  and  Marine  ATC  students  (7251 MOS)  on  19  different 
individual  traits.  Through  both  concurrent  and  predictive  validity  designs,  scores  on  the 
NCAPS  were  correlated  with  measures  of  schoolhouse  academic  performance  and 
Operational  Forces  7257  qualification.  The  results  of  hierarchical  regression  analyses 
indicated  that  a  subset  of  relevant  NCAPS  traits  were  statistically  significant  predictors 
of  schoolhouse  performance  and  7257  qualification.  Two  separate  predictive  profiles 
were  developed.  These  profiles  were  referred  to  as  the  Marine  Air  Traffic  Control 
Suitability  Screening  Test  Passing  (MATC-ST  Passing  and  7257  Qualification)  and  the 
Marine  Air  Traffic  Control  Suitability  Screening  Test  Qual  (MATC-ST  7257 
Qualification).  NOTE-  At  the  time  of  this  report,  the  prototype  cognitive  test  was 
developed,  pilot  tested,  and  in  its  Beta  form. 

Findings  or  Results 

The  subset  of  NCAPS  traits  that  demonstrated  statistically  significant  prediction  for 
ATC  performance  Passing/7257  Qualification  and  thus  were  included  the  Marine  Air 
Traffic  Controller  -  Suitability  Screening  Test  (Passing)  algorithms  were:  Innovation, 
Social  Orientation,  Perceptiveness,  Social  Orientation,  and  Tolerance  for  Ambiguity. 
Hierarchical  multiple  regression  analyses  suggest  that  the  non-cognitive  suitability 
measure  adds  incremental  validity  above  and  beyond  the  ASVAB  score,  and  accounts  for 
8.6%  of  the  variance  of  ATC  schoolhouse  performance/7257  Qualification  (R2=.293,  (3  = 
.697,  t  =  3.070  p  <  .000).  Given  a  MATC-ST  score,  one  can  predict  the  expected  level  of 
success  an  individual  Marine  is  likely  to  exhibit  in  the  ATC  schoolhouse  and  7257 
Qualification  in  order  to  make  subsequently  valid  ATC  duty  selection  decisions.  NOTE- 
Data  collection  was  underway  for  the  prototype  cognitive  ability  measures  at  the  time  of 
this  report. 

Conclusions 

The  Marine  ATC  Suitability  Screening  Test  is  a  valid  algorithm  derived  from 
dimensions  of  NCAPS  and  has  been  shown  to  be  predictive  of  success  for  Marine  ATCs 
in  the  Schoolhouse  and  7257  Qualification  in  the  Operating  Forces.  Those  with  higher 
suitability  scores  performed  better  on  knowledge  and  simulation  exams  and  graduated 
at  a  higher  rate  than  those  with  low  suitability  scores,  as  well  as  qualifying  as  a  7257 
ATC.  Data  collection  and  analysis  from  the  ATC  schoolhouse  graduation  to  7257 
qualification  provided  confirmation  of  the  MATC-ST’s  predictive  validity,  both  for 
training  performance  and  ATC  job  performance.  Further  analysis  to  derive  conservative 
cut-scores  from  the  combined  results  of  the  concurrent  and  predictive  validation  from 
the  ATC  Schoolhouse/Operating  performance  is  completed.  If  the  noncognitive  MATC- 
ST  is  to  be  utilized  for  selection/classification  it  is  recommended  that  it  be  used  only  in 
addition  to  the  ASVAB,  and  implemented  at  the  Military  Entrance  Processing  Station 
(MEPS)  or  in  the  pipeline  prior  to  a  Marine  ATC  MOS  designation.  Taken  together,  the 
two  tests  are  predicted  to  reduce  attrition/revocation  and  increase  the  quality  of 
Marines  selected  for  the  ATC  MOS.  MATC-ST  implementation  can  improve  the  Marine 
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ATC  screening  process  through  better  schoolhouse  performance,  better  operating  forces 
performance,  less  attrition/revocation  costs,  and  increased  diversity  through  fair,  valid 
screening  improvements. 
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Introduction 


The  previous  technical  reports  titled  Suitability  Screening  Test  for  Marine  Corps  Air 
Traffic  Controllers  Technical  Report  Phase  I  &  Phase  II  (Walker,  Farmer,  &  Roberts, 
2013a;  Walker,  Farmer,  &  Roberts  2013b),  gave  a  complete  introduction  of  the  research 
involved  for  this  study.  As  a  review,  the  current  Air  Traffic  Control  selection  process 
utilized  by  the  US  Marine  Corps  is  being  reviewed  as  a  result  of  an  unacceptable  number 
of  Marine  Air  Traffic  Controllers  (ATCs)  being  classified  as  “unsuited”  for  ATC  duty  as 
defined  by  performance  and  behavioral  expectations.  The  first,  most  important  step  in 
creating  a  more  efficient  force  is  to  improve  the  Air  Traffic  Controller  (ATC) 
screening/selection  process.  This  research  effort  focused  on  developing  and  validating  a 
non-cognitive  and  cognitive  Air  Traffic  Controller  profile  that  could  be  incorporated  into 
a  model  for  their  selection.  Such  a  profile  would  allow  the  Marine  Corps  to  identify  those 
Marines  who  are  likely  to  be  successful  as  Air  Traffic  Controllers,  those  who  need  a 
greater  depth  of  screening,  and  those  who  are  likely  to  fail. 

During  Phase  I  and  II  of  this  research  study  the  focus  was  on  data  collection  and 
analysis  from  Marine  ATCs  in  the  operating  forces  and  within  the  ATC  schoolhouse. 
Phase  I  results  reported  significant  findings  within  the  Operating  Forces,  and 
recommended  further  predictive  validity  analysis  from  the  ATC  schoolhouse  data.  Phase 
II  reported  significant  findings  within  the  ATC  schoolhouse,  and  recommended  further 
predictive  validity  analysis  following  ATC  schoolhouse  Marines  to  7257  MOS 
qualification.  This  report  will  discuss  Phase  III  of  the  study  and  includes  the  predictive 
validity  analysis  from  the  ATC  schoolhouse  and  7257  MOS  qualification  data. 

Background 

Phase  I  of  this  study  was  a  concurrent  validity  study  utilizing  data  collection  and 
analysis  from  Marine  ATCs  in  the  operating  forces.  The  subset  of  NCAPS  traits  that 
demonstrated  statistically  significant  prediction  for  ATC  (operating  forces)  performance 
and  were  included  the  Marine  Air  Traffic  Controller  -  Suitability  Screening  Test  (MATC- 
ST)  algorithms  were:  Adaptability /Flexibility,  Vigilance ,  Empathy,  and  Self-Reliance. 
Hierarchical  regression  analyses  suggest  that  the  non-cognitive  suitability  measure  adds 
incremental  validity  above  and  beyond  the  ASVAB  score,  and  accounts  for  14%  of  the 
variance  of  ATC  job  performance  (R2=.i4i,  (3=-375,  t=5.804,  p<.ooo).  Given  this  MATC- 
ST  score,  one  can  predict  the  expected  level  of  success  an  individual  Marine  is  likely  to 
exhibit  on  the  job  and  can  make  subsequently  valid  selection  decisions. 

Phase  II  of  this  study  was  a  predictive  validity  study  utilizing  data  collection  and 
analysis  from  Marine  ATCs  in  the  schoolhouse.  There  were  two  subsets  of  NCAPS  traits 
that  demonstrated  statistically  significant  prediction  for  ATC  (schoolhouse) 
performance  and  were  included  the  Marine  Air  Traffic  Controller  -  Suitability  Screening 
Test  (MATC-ST)  algorithms.  The  MATC-ST  (A):  Achievement  Motivation,  Initiative, 
Perceptiveness/Depth  of  Thought,  Social  Orientation,  and  Self-Reliance;  and  the 
MATC-ST  (B):  Achievement  Motivation,  Empathy,  Perceptiveness/Depth  of  Thought, 
Self-Reliance,  and  Vigilance.  The  MATC-ST  B  was  included  because  it  incorporated 
dimensions  that  overlap  in  content  to  the  significant  predictors  for  operational 
performance  developed  in  Phase  I  of  this  research  project.  Hierarchical  regression 
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analyses  suggest  that  the  non-cognitive  suitability  measure  adds  incremental  validity 
above  and  beyond  the  ASVAB  score,  and  accounts  for  29%  of  the  variance  of  ATC 
schoolhouse  performance  (R2=.2o8,  P  =  .541,  t  =  4.646,  p  <  .000).  Given  these  MATC- 
ST  scores,  one  can  predict  the  expected  level  of  success  an  individual  Marine  is  likely  to 
exhibit  in  the  ATC  schoolhouse  and  can  make  subsequently  valid  ATC  duty  selection 
decisions. 

Phase  III  of  this  study  will  discuss  findings  from  validity  analysis  utilizing  Marine 
ATC  schoolhouse/7257  MOS  qualification  passing  and  7257  Qualification  data.  This 
effort  considers  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  standard  that 
is  established  to  minimize  ATC  academically  related  training  failures  in  conjunction 
with  the  suitability  screening  test  to  replicate  Phase  I  &  II  findings.  In  addition  to  the 
noncognitive  testing  Phase  III  also  included  the  development  and  initial  pilot  testing  of 
the  Prototype  Marine  ATC  Cognitive  Test. 


Method 

The  validation  approach  chosen  for  this  project  was  a  criterion-related  strategy.  This 
is  traditionally  accomplished  by  obtaining  the  test  scores  of  job  applicants  as  predictors 
and  then  collecting  measures  of  these  same  individuals’ job  performance,  the  criterion 
(or  criteria  if  more  than  one  type  of  measure  is  collected).  The  predictor  test  scores  are 
then  related  statistically  to  how  well  individuals  perform  on  the  job  and,  if  successful, 
can  be  used  to  identify  the  most  qualified  candidates  for  the  position  by  predicting  how 
individuals  with  particular  test  scores  will  likely  perform.  This  validation  methodology  is 
one  of  three  validation  strategies  presented  in  the  Uniform  Guidelines  on  Employee 
Selection  Procedures  (1978,  EEOC),  the  Standards  for  Educational  and  Psychological 
Testing,  and  the  Society  for  Industrial/  Organizational  Psychology’s  (SIOP)  Principles 
for  the  Validation  and  Use  of  Personnel  Selection  Procedures  (1987). 

In  personnel  selection  and  classification  it  is  customary  to  develop  measures  that 
predict  job  performance  and/or  job  tenure.  Measures  given  to  job  applicants  need  to 
assess  the  knowledge,  skills,  and  abilities  necessary  for  successful  performance  in  a 
particular  job,  ideally  without  producing  adverse  impact  (large  mean  differences)  for 
racial,  ethnic,  or  gender  groups.  This  study  relied  on  a  mix  of  predictive  and  concurrent 
validity  designs  using  NCAPS  and  Marine  ATC  performance  measures.  A  concurrent 
validity  design  is  when  the  predictor  is  administered  to  participants  close  in  time  to 
when  the  criterion  (performance  measure)  is  collected  (e.g.,  on  the  job).  NCAPS  was 
administered  in  such  a  concurrent  validity  design  to  individuals  who  are  already  ATCs. 

A  predictive  validity  design  is  when  the  predictor  is  administered  long  in  advance  to 
all  applicants  who  have  yet  to  be  selected  for  the  job,  and  thus  represents  a  more 
realistic  setting  in  which  the  instrument  will  ultimately  be  used.  Additionally,  NCAPS 
was  administered  in  such  a  predictive  validity  design  to  individuals  who  have  not  yet 
been  trained  as  ATCs.  The  ASVAB  (GT  Score)  was  also  administered  in  a  predictive 
validity  design. 

In  order  to  statistically  perform  this  validation  for  Phase  III,  measurements  of  job 
performance  and  NCAPS  scores  were  provided  for  Marines  currently  training  at  the 
Marine  ATC  Schoolhouse  and  those  Marines  who  have  qualified  as  7257  MOS  (both 
concurrent  and  predictive  data).  This  included  Marine  ATCs  already  attending  the  ATC 
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schoolhouse,  any  new  Marine  ATC  arrivals  at  the  schoolhouse  and  Marines  attaining  or 
revocating  from  the  7257  MOS. 

Cognitive  Prototype  Development 

Phase  III  required  the  development  of  a  cognitive  ability  measures  for  Marine  ATCs 
similar  in  nature  to  those  used  by  the  Federal  Aviation  Administration.  As  a  first  step, 
the  authors  evaluated  the  FAA’s  Air  Traffic  Scenario  Test  (AT-ST).  Based  on  this 
evaluation  and  a  review  of  the  Marine  Air  Traffic  Control  Training  and  Revocations 
Study  (Northrop  Grumman  Information  Systems,  2011),  the  authors  decided  to  build  a 
battery  of  tests  to  measure:  1)  working  memory  capacity,  2)  executive  control 
functioning,  3)  perceptual  speed,  4)  multitasking  ability,  and  5)  inductive  reasoning 
ability. 

A  working  memory  capacity  test  was  developed  because  working  memory  has  been 
linked  to  performance  on  complex  jobs  in  a  number  of  domains  (e.g.,  De  Dreu,  Nijstad, 
Baas,  Wolsink,  &  Roskes,  2012).  Further,  executive  control  has  been  shown  to  similarly 
predict  complex  performance  (Kane  &  Engle,  2003).  A  perceptual  speed  test  was 
developed  due  to  the  perceived  demands  on  ATCs  to  track  visually  displayed  data  over 
time  in  potentially  high-density  task  environments  (Ackerman  &  Cianciolo,  2000). 
Multitasking  ability  was  likewise  targeted  due  to  the  perception  that  ATCs  must  process 
information  near-simultaneously  for  multiple  aircraft  (Hambrick,  Oswald,  Darowski, 
Rench,  &  Brou,  2010).  Finally,  the  role  of  strategic  thinking  in  performing  ATC  tasks 
appeared  critical,  and  previous  studies  linked  inductive  reasoning  ability  with  strategy 
development  (Schunn  &  Reder,  2001). 

With  these  abilities  identified,  a  set  of  tests  was  developed  that  attempted  to  measure 
them  both  in  isolation  and  in  tandem.  Four  tests  were  developed  for  the  final  prototype 
battery.  First,  “assessment  1”  was  developed  to  test  working  memory  capacity  and 
executive  control  functioning  (as  defined  by  resistance  to  interference).  The  test  has  four 
phases.  During  the  first  phase,  users  are  presented  with  a  set  of  to-be-remembered 
stimuli  one  item  at  a  time.  After  studying  an  item,  the  user  must  click  a  button  to  see  the 
next  item.  When  all  items  have  been  presented,  the  user  is  prompted  to  enter  the  list  of 
stimuli  in  order.  Successful  recall  results  in  a  longer  stimulus  set  being  presented,  and 
users  continue  to  view  and  recall  stimuli  until  they  can  no  longer  successfully  recall  lists 
in  order.  This  process  is  repeated  for  spatial  stimuli,  verbal  stimuli,  and  numerical 
stimuli.  The  stimuli  with  which  the  user  is  most  successful  is  used  in  later  phases  of  the 
test,  and  the  average  time  spent  looking  at  items  within  the  stimulus  set  is  also  used  to 
tailor  the  later  phases  of  the  test  to  the  individual  user.  During  Phase  II,  the  user  is 
presented  with  processing  tasks  to  complete.  Spatial  processing  tasks  involve 
performing  mental  rotation  to  determine  if  one  image  is  the  same  as  another  which  has 
been  rotated  by  60, 120, 180,  240,  or  300  degrees.  Verbal  processing  tasks  involve 
reading  a  sentence  and  deciding  if  it  is  grammatically  and  structurally  correct  or 
scrambled.  Numerical  processing  tasks  involve  viewing  an  equation  such  as  “21  +  13-4 
=  29”  and  indicating  if  the  equation  is  true  or  false.  Users  respond  to  processing  tasks 
until  they  have  correctly  responded  to  five  of  each  type.  The  average  time  required  to 
accurately  respond  to  processing  items  is  recorded  and  used  to  tailor  the  test  in  the 
remaining  phases.  Phase  III  integrates  the  first  two  phases  by  presenting  a  set  of  to-be- 
remembered  items  between  processing  tasks  (e.g.,  “2”  is  presented  as  a  to-be- 
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remembered  item,  then  a  spatial  processing  task  is  displayed,  next  a  “4”  is  presented  as 
a  to-be-remembered  item,  etc.).  The  to-be-remembered  set  is  composed  of  the  stimulus 
type  identified  during  Phase  I,  and  each  item  is  presented  for  the  average  study  time 
required  in  Phase  I.  The  processing  tasks  are  chosen  randomly  from  the  two  stimulus 
types  that  are  different  from  the  to-be-remembered  set’s  stimulus  type,  and  a  time-limit 
is  imposed  equal  to  two  standard  deviations  above  the  average  processing  time 
identified  in  Phase  II.  Like  Phase  I,  successful  recollection  of  a  stimulus  set  results  in  a 
longer  set  being  presented,  and  users  stop  when  they  can  no  longer  successfully  recall 
the  set  in  order.  In  addition,  phase  three  requires  that  at  least  two-thirds  of  the 
processing  tasks  be  answered  correctly  within  a  set  before  advancement  to  a  longer  set  is 
allowed.  Phase  IV  is  identical  to  Phase  III  except  that  the  processing  tasks  all  come  from 
the  stimulus  type  that  matches  the  to-be-remembered  stimulus  set’s  type.  This  is 
believed  to  result  in  additional  cognitive  interference  and  therefore  require  more 
executive  control  to  maintain  performance  levels.  See  the  Appendix  I  for  further  details. 

“Assessment  two”  was  developed  to  test  perceptual  speed  and  multitasking  ability. 
The  test  presents  multiple  objects  consisting  of  an  identifier  label  and  a  parameter  value 
which  move  randomly  within  a  portion  of  the  screen.  The  parameter  value  of  each  object 
must  be  compared  to  a  range  of  values  that  is  acceptable.  This  range  of  acceptable  values 
changes  over  time,  requiring  users  to  continuously  scan  not  only  the  objects,  but  also  the 
acceptable  ranges  for  the  objects’  parameter  values.  When  a  user  detects  that  an  object’s 
parameter  value  falls  outside  of  the  acceptable  range,  he  or  she  must  press  a  button  to 
report  the  contact.  Reporting  the  contact  requires  the  user  to  remember  and  enter  the 
object’s  identifier  label.  Signal-detection  theory  analysis  is  used  to  determine  the  d’ 
(pronounced  d-prime)  sensitivity  of  users  as  they  scan  the  screen  for  out-of-bound 
values.  In  addition  to  the  object  scanning  demands  of  the  task,  another  section  of  the 
screen  presents  the  users  with  a  guidable  circle  that  must  be  navigated  over  a  set  of 
targets  using  the  arrow  keys  on  the  keyboard  or  screen.  Users  get  points  for  successfully 
navigating  over  the  targets.  The  points  acquired  for  this  portion  of  the  task  combined 
with  the  d’  sensitivity  acquired  from  the  first  portion  of  the  task  provides  a  measure  of 
user  multitasking  ability.  This  portion  of  the  test  also  serves  as  a  color-blindness 
indicator,  as  the  targets  vary  in  color  in  such  a  way  that  various  types  of  colorblindness 
will  render  users  unable  to  detect  the  location  of  the  targets  against  the  colored 
background.  Thus,  drops  in  performance  for  color-blind-sensitive  targets  compared  to 
color-blind-insensitive  targets  provides  an  indication  of  user  color-blindness.  See  the 
Appendix  I  for  further  details. 

“Assessment  three”  was  developed  to  measure  inductive  reasoning  ability  which  has 
been  linked  to  strategy  development  and  analogical  reasoning.  The  first  part  of  the  test 
presents  the  user  with  a  set  of  puzzles  consisting  of  an  eight  by  eight  grid  populated  in 
part  with  a  series  of  “Xs”  and  check  marks.  A  set  of  colored  circles  begin  suspended 
above  the  top  row  of  the  grid.  Without  intervention  from  the  user,  the  circles  will  move 
downward  towards  the  bottom  of  the  grid  until  they  impact  an  “X,”  resulting  in  a  failed 
trial.  The  user  may  intervene  in  the  path  taken  by  the  circles  by  placing  shapes  on  the 
grid  that  cause  the  colored  circles  to  begin  moving  in  new  directions.  Each  available 
shape  has  a  different  effect  on  the  circles  as  a  function  of  their  color.  This  means  that  to 
successfully  guide  the  circles  to  check  marks  (the  goal  of  the  puzzle),  users  must  place 
the  shapes  on  the  grid  in  such  a  way  that  all  balls,  regardless  of  color,  land  on  check 
marks  without  hitting  “Xs.”  The  second  part  of  the  test  presents  users  with  a  series  of 


4 


analogies  to  solve  by  either  selecting  a  word  to  complete  partial  analogy  being  displayed, 
or  selecting  true  or  false  to  full  analogies  being  displayed.  See  the  Appendix  I  for  further 
details. 

“Assessment  four”  was  developed  to  measure  each  of  the  skills  captured  in 
assessments  l  -  3  in  tandem.  The  test  uses  a  “manufacturing  plant”  design  wherein  users 
must  respond  to  orders  for  various  types  of  merchandise.  The  plant  consists  of  five 
different  machines  each  producing  parts  of  the  types  of  merchandise  for  sale,  as  well  as 
a  warehouse  for  storing  parts  and  completed  products.  Because  the  machines  take 
different  amounts  of  time  to  produce  their  parts,  the  users  must  strategically  plan  how 
they  operate  the  plant  to  avoid  bottleneck  problems.  They  must  also  do  their  best  to 
maintain  a  small  inventory,  as  storage  costs  money  and  eats  into  the  profit  they  would 
otherwise  make  from  the  sale  of  merchandise.  See  the  Appendix  I  for  further  details. 


Data  Collection 

Data  were  collected  from  Marine  ATC  students  (7251 MOS)  at  the  Air  Traffic 
Controller  Schoolhouse  in  Pensacola,  Florida  and  from  Operational  Forces  Marine 
7257s  in  training  and  those  that  have  qualified  during  the  course  of  the  study. 
Operational  Forces  qualification  data  was  collected  for  nine  Air  Traffic  Control  Facilities 
(ACTFs):  Camp  Pendleton,  Beaufort,  Miramar,  Cherry  Point,  New  River,  Yuma, 
Futenma,  Iwakuni,  and  Quantico.  Data  collection  consisted  of  interviews,  observations, 
performance  measurements,  and  NCAPS  scores.  The  interviews  and  observations  were 
covered  thoroughly  in  Phase  I  &  II  technical  reports  (Walker,  Farmer,  &  Roberts,  2012). 
This  technical  report  will  focus  on  additional  data  collection  efforts  unique  to  Phase  III 
of  the  study. 

ATC  Schoolhouse 

The  14-week  school  consists  of  three  blocks  of  training  that  have  knowledge  based 
written  tests  and  performance  based  assessments.  Block  One  training  is  six  weeks  long, 
entirely  knowledge  based,  and  concludes  with  the  FAA  qualification  written  exam.  Block 
Two  consists  of  local  control,  ground  control,  and  flight  data  performance  based 
assessments.  Block  Three  consists  of  basic  radar,  ASR,  PAR,  and  Arrival  written  tests 
and  performance  based  assessments. 

The  schoolhouse  requires  a  70%  or  higher  grade  to  pass,  however  in  the  operating 
forces  an  80%  or  higher  is  required  to  receive  qualifications.  ATC  officials  view  ATC 
school  as  preliminary  and  expect  that  more  extensive  learning  occurs  with  On-the- Job- 
Training  (OJT),  the  platform  MOS,  and  other  required  operating  force  qualifications  to 
maintain  qualified  controller  status. 

•  Marine  Corps  ATC  has  an  ASVAB  GT  (VE,  Verbal  +  AR,  Arithmetic  Reasoning  + 
MC,  Mechanical  Comprehension)  score  of  no  as  a  minimum  qualifier.  A  Marine 
recruit  is  then  designated  into  Aviation  and  disseminated  between  ATC,  Weather, 
and  UAV.  Instructors  also  confirmed  that  this  process  leads  to  motivational 
concerns  as  to  whether  Marines  selected  for  ATC  School  have  the  motivation  to 
be  there. 
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Schoolhouse  Performance  Measures 

In  order  to  measure  performance  at  the  ATC  schoolhouse  we  collected  Grade  Point 
Averages  (GPA)  for  each  Marine  student  at  each  block  of  training.  Completion  of  the 
schoolhouse  Blocks  1-3  training  resulted  in  a  schoolhouse  passing  score  of  1  and  failure 
at  any  Block  1-3  resulted  in  a  schoolhouse  passing  score  of  o. 

NCAPS  at  Schoolhouse 

The  Navy  Computer  Adaptive  Personality  Scales  (NCAPS)  is  a  web-based  non- 
cognitive  measurement  tool  consisting  of  19  dimensions.  NCAPS  was  administered  to 
incoming  and  current  7251s. 

MCAS  ATC  Operational  Forces 

During  Phase  I  &  II,  after  reviewing  the  existing  job  analysis  information,  sites  visits 
were  conducted  to  observe  controllers  from  the  various  MCASs.  The  primary  purpose  of 
these  initial  site  visits  was  to  gain  a  better  understanding  of  the  ATC  job,  the  on-the-job 
training  at  each  of  the  MCASs,  and  to  speak  with  SMEs  and  supervisors  about  the 
training  process.  Observations  were  made  of  the  controllers  from  various  radar  and 
tower  positions  performing  their  job,  and  researchers  discussed  the  various  components 
of  the  job  and  training  process  with  the  controllers,  their  trainers,  and  supervisors.  In 
some  cases,  supervisors  identified  high  and  low  performers  within  their  teams  and 
described  qualities  or  abilities  necessary  for  high  performing  ATC  Marines. 

When  a  7251  arrives  to  a  MCAS  ATC  Facility  (ATCF)  they  endure  an  indoctrination 
process  that  consists  of  an  assignment  to  an  On-the-job  Instructor  (OJTI),  testing,  and 
orientation  syllabus.  Each  MCAS  ATCF  has  an  indoctrination  process  to  help  a  7251 
understand  the  process  to  7257  qualification  at  that  particular  ATCF.  Each  MCAS  ATCF 
also  has  permanent  qualified  civilian  ATCs  as  training  specialists  to  ensure  continuity. 

Phase  III  data  collection  from  MCAS  ATC  Operational  Forces  included  time  to  7257 
qualification  and  7257  GPA  if  available.  Successful  qualification  of  7257  received  a  score 
of  1  and  unsuccessful  completion  (failure)  or  revocation  received  a  score  of  o. 

Cognitive  Prototype  Test 

Phase  III  consisted  of  the  development  of  the  cognitive  prototype  and  the  initial  pilot 
testing  of  the  four  assessment  measures.  Data  collection  was  conducted  for  all  seven 
MCAS  ATCFs  from  Phase  I  &  II  on  all  4  assessments.  At  the  time  of  this  report,  data  was 
still  being  collected  and  will  be  analyzed  in  the  near  future. 

Validity  Analysis 

Validity,  for  this  study,  was  defined  by  the  correlation  between  a  predictor  of  success 
(such  as  the  ASVAB  GT  Score,  or  NCAPS)  and  an  objective  performance  measure  (such 
as  school  grades  or  graduation  status,  or  time  to  7257  qualification).  The  ASVAB  (GT 
Score)  has  demonstrated  validity  for  predicting  job  performance,  but  is  most  predictive 
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of  academic  performance.  NCAPS  has  demonstrated  validity  for  predicting  job 
performance.  The  two  predictors  generally  do  not  correlate,  but  it  depends  upon  the 
constructs  being  measured  by  the  specific  criterion. 

Performance  Measures 

Job  performance  is  a  multifaceted  construct  that  can  be  measured  in  a  variety  of 
ways  (peer  ratings,  supervisor  ratings,  customer  ratings,  output,  qualifications,  absence 
of  errors,  etc.).  For  Phase  III,  a  variety  of  proxies  for  job  performance  were  considered, 
but  the  following  provided  the  most  significant  results. 

Schoolhouse  performance  measures: 

GPA  scores  for  Block  One,  Block  Two,  Block  Three,  and  Blocks  One  through  Three 
(1-3)  Cumulative  GPA  were  collected  and  standardized.  ASVAB  (GT  Scores)  were  also 
collected. 

MCASATC  Operational  Forces  performance  measures: 

While  some  Marines  are  able  to  master  the  position  and  pass  the  qualification  exams 
within  a  few  months,  others  take  much  longer.  Marines  who  are  able  to  meet 
qualification  at  a  quicker  pace  are  often  considered  better  performers  and  more 
proficient  controllers.  Therefore,  time  to  qualify  may  be  considered  a  proxy  for  job 
performance,  in  that  these  individuals  are  more  naturally  suited  to  the  demands  of  ATC 
positions,  and  thus  require  less  time  to  exhibit  proficient  performance. 

Time  to  qualify  was  calculated  from  training  start  date  to  qualification  date  for  7257, 
7252,  and  7253/54  and  was  standardized  across  ATCFs.  Each  ATCF  provides  a  unique 
training  opportunity  for  the  Marines  stationed  there;  and  as  a  result,  Marines  training  at 
different  bases  are  exposed  to  overlapping,  but  not  quite  parallel  training  opportunities. 
For  example,  a  station  that  controls  a  complex  air  space  that  can  facilitate  a  variety  of 
aircraft  operations  may  provide  opportunity  for  Marines  to  train  on  a  range  of  real- 
world  scenarios.  Meanwhile,  Marines  at  bases  with  less  traffic  and  less  complexity  in 
airspace  are  not  exposed  to  these  challenges  as  readily  or  consistently.  This  variability  in 
exposure  is  likely  to  facilitate  training  on  some  bases,  and  allow  Marines  to  more  quickly 
qualify  on  their  positions.  Given  this  set  of  circumstances,  standardization  was 
necessary  for  time  to  qualify  across  ATCFs,  such  that  those  who  quickly  qualify  at  their 
respective  stations  will  be  acknowledged  as  high  performing  trainees. 

There  was  considerable  variability  among  each  ATCF;  therefore  a  standardization 
technique  was  used  to  place  performance  scores  by  ATCF  on  a  common  scale.  For 
example,  a  high  performer  at  Cherry  Point  ATCF  may  qualify  in  four  months  whereas  a 
high  performer  at  Yuma  ATCF  may  qualify  in  six  months.  In  this  case,  standardized 
scores  per  ATCF  will  recognize  high  performers  across  various  units  and  not 
differentiate  them  when  site  factors,  not  personnel  attributes  or  ability,  account  for  the 
time  differentials. 

Marines  attaining  the  7257  qualification  received  a  score  of  1  and  those  who  did  not 
attain  the  7257  MOS  due  to  poor  performance  received  a  score  of  o.  Medical  drops  and 
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Non-Academic  drops  were  not  included  in  analysis.  Qualification  test  scores  (written 
exam  and  performance  exam)  were  collected  and  averaged  for  a  GPA  for  7257.  The 
GPAs  were  also  standardized  by  ATCF. 


Results 

The  resulting  data  from  the  ATC  schoolhouse  and  Operational  forces  were  cleaned 
and  subjected  to  various  analyses  including  validity  analysis.  This  section  describes  the 
data  obtained  and  analyses  with  the  results  reported  up  front.  Validity  analysis 
determined: 

•  NCAPS  scores  are  a  valid  predictor  of  success  for  Marine  Air  Traffic  Controllers 
passing  the  schoolhouse  and  attaining  7257  qualification.  Success  in  the 
schoolhouse  is  determined  by  graduation  from  ATC  School.  Success  in  the 
Operating  Forces  is  determined  by  performance  at  a  Marine  ATCF  and  7257 
qualification. 

•  ASVAB  GT  Scores  correlated  significantly  with  Passing  for  the  schoolhouse  as  did 
NCAPS  measures. 

•  Regression  analyses  clearly  indicate  that  non-cognitive  measures,  especially  those 
assessed  with  the  Marine  Air  Traffic  Control  Suitability  Test  Passing  and  Qual 
(MATC-ST  Passing  and  Qual):  add  incremental  validity  above  the  ASVAB  GT 
Score. 

Background  and  Demographics 

This  section  describes  the  sample  of  participants  who  took  part  in  the  study,  as  well 
as  the  characteristics  of  the  performance  measures  collected  from  those  participants. 
Table  1  presents  background  and  demographic  information  for  405  Marine  Corps  ATC 
(7251/7257)  participants.  The  majority  of  the  sample  was  male  (92.6%),  White  (67%), 
with  less  than  4  years  of  service  (93%).  Most  of  the  sample  was  Private  First  Class  (PFC; 
E2  paygrade),  (72.8%). 
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Table  1 

Marine  Corps  ATC  Demographics 


Frequency 

Percent 

Gender 

Male 

375 

92.6 

Female 

30 

7  A 

White 

271 

66.9 

Hispanic 

62 

15.3 

Ethnicity 

Black 

37 

9.1 

Asian  or  Pacific  Islander 

15 

3.7 

American  Indian 

5 

1.2 

Other 

15 

3.8 

El  (PVT) 

42 

10.4 

E2  (PFC) 

295 

72.8 

Rank 

E3  (LCPL) 

19 

4.7 

E4  (CPL) 

28 

6.9 

E5  (SGT) 

21 

5.2 

YoS 

<4 

379 

93.6 

5-10 

26 

6.3 

NCAPS  Data  Summary 

At  the  time  of  this  report,  complete  data  was  obtained  from  the  NCAPS  measure  for 
405  Marine  Corps  ATC  (7251/7257  MOS)  participants. 

NCAPS  is  a  computerized  adaptive  personality  measure  consisting  of  19  components, 
four  of  which  were  related  to  attributes  relevant  to  the  current  research  effort.  For  each 
NCAPS  component,  the  data  consist  of  a  trait  level  value  that  describes  the  participants 
standing  on  the  construct  being  measured,  and  a  PSD  or  posterior  standard  deviation, 
which  is  an  index  of  algorithm  convergence.  A  theta  value  is  defined  as  an  individual’s 
standing  on  a  construct  being  measured  using  adaptive  testing  and  item  response 
theory,  where  items  are  presented  to  individuals  based  on  their  responses  to  prior  items. 
This  process  continues  until  the  algorithm  closes  in  on  that  individual’s  standing  on  the 
construct,  which  is  represented  by  the  value  of  theta. 

There  were  seven  relevant  NCAPS  dimensions  for  the  ATC  Passing  and  7257 
Qualification.  These  NCAPS  dimensions  were  used  to  develop  the  algorithms  for  the 
Marine  Air  Traffic  Control  Suitability  (MATC-ST).  There  are  two  relevant  algorithms  for 
the  ATC  suitable  performance.  These  will  be  referred  to  as  the  Marine  Air  Traffic 
Control  Suitability  Test  Passing  (MATC-ST  P)  and  the  Marine  Air  Traffic  Control 
Suitability  Test  Qualification  (MATC-ST  Q).  The  Marine  Air  Traffic  Control  Suitability 
Test  Passing  dimensions  were:  Tolerance  for  Ambiguity,  Innovation, 
Perceptiveness/Depth  of  thought,  and  Social  Orientation.  The  Marine  Air  Traffic 
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Control  Suitability  Test  Qualification  dimensions  were:  Adaptability/Flexibility,  Positive 
Self-Concept,  and  Vigilance.  The  MATC-ST  Q  has  a  much  smaller  sample  size  as  a  result 
of  the  9-month  MARADMIN  requirement  for  7257  qualification.  The  degree  of 
relevance,  or  relatedness  (validity)  was  derived  through  regression  analyses,  which 
identified  these  dimensions  (or  traits)  as  statistically  significant  predictors  of  ATC 
performance  at  the  Schoolhouse  and  7257  qualification. 

ATC  Performance  Data  Summary 

The  ATC  performance  data  summary  in  Table  2  includes  the  minimums,  maximums, 
means,  modes,  and  standard  deviations  of  GT  Score,  Passing,  and  7257  Qualification. 

GT  Scores  were  collected  for  404  ATC  Marines,  with  a  minimum  GT  Score  of  105,  a 
maximum  GT  Score  of  146,  a  mean  GT  Score  of  118.45,  and  the  standard  deviation  for 
GT  Score  was  7.559. 

Passing  was  collected  for  321  ATC  Marines,  with  212  passing  and  109 
failures/revocations.  Passing  represented  66.0%  and  the  failure  percentage  was  34%. 
Qualification  was  collected  for  114  ATC  Marines,  with  100  qualified  7257s  and  14 
revocations.  Qualification  represented  87.6%  and  revocation  represented  12.4%.  (The 
Qualification  sample  size  is  smaller  as  a  result  of  the  9-month  MARADMIN  requirement 
for  7257  qualification). 


Table  2 

ATC  Performance  Descriptive  Statistics 


N 

Minimum 

Maximum 

Mean 

Std.  Deviation 

GT  Score 

404 

105 

146 

118.45 

7.559 

Passing  * 

321 

34.0%* 

66.0%* 

N/A 

N/A 

Qualification** 

114 

12.4%* 

87.6%* 

N/A 

N/A 

*Passing  percentages  include  passing  and  failures/revocations. 
^^Qualification  percentages  include  7257  qualification  and  revocations. 


Marine  ATC  Suitability  Test  Score  and  Validity  Analysis 

The  Marine  ATC  Suitability  Test  Passing  (MATC-ST  P)  and  MATC-ST  Qualification 
(MATC-ST  Q)  score  for  the  ATC  population  is  calculated  from  dimensions  of  NCAPS 
that  were  submitted  in  regression  analysis  with  the  ATC  schoolhouse  Block  1-3 
completion  (Passing)  and  7257  Qualification  as  the  performance  criterion.  This  section 
describes  the  correlation  and  regression  analyses. 

Table  2.2  presents  the  intercorrelations  of  the  predictor  test  scores,  (MATC-ST  P), 
(MATC-ST  Q),  ASVAB  GT  and  Passing/Qualification. 
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Table  3 

Predictor  Test  Scores  and  Passing/Qualification  Intercorrelations 


MATC-ST  P 

MATC-ST  Q 

GT  Score 

Passing 

7257  Qual 

MATC-ST  P 

1 

MATC-ST  Q 

142** 

1 

GT  Score 

.173** 

.113* 

1 

Passing 

288** 

.053** 

.296** 

1 

7257  Qualification 

.178 

.416** 

.045 

1.00** 

.1 

*  Correlation  is  significant  at  the  0.05  level  (2-tailed). 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 

Note:  The  Qualification  sample  size  is  smaller  as  a  result  of  the  9-month  MARADMIN  requirement  for 
7257  qualification 

Table  3  shows  the  intercorrelations  between  GT  Score,  the  MATC-ST  P  and  MATC- 
ST  Q,  and  the  Marine  ATC  performance  metrics.  Predictor  test  scores  were  significantly 
correlated  with  one  another,  Suitability  P  and  Q  algorithms,  and  with  GT  Score  at  the 
p<.oi  level;  with  exception  for  correlations  with  7257  Qualification  with  MATC-ST  P  and 
GT  Score).  GT  Score  was  also  significantly  correlated  to  MATC-ST  P  and  MATC-ST  Q  at 
the  p<.05  and  p<.oi  levels  respectively.  For  completeness,  the  data  were  used  in  a 
hierarchical  regression. 

Through  hierarchical  regression,  the  prediction  of  the  MATC-ST  P  and  Q  score  on 
performance  outcomes  may  be  isolated  and  compared  to  the  prediction  of  other  factors, 
such  as  the  ASVAB  GT  score,  as  well  as  the  prediction  of  composites  of  multiple  factors 
at  once.  This  technique  uses  comparisons  of  successive  regression  models  and 
determines  the  significance  that  each  one  has  above  and  beyond  the  others.  Three 
models  were  tested  and  compared:  (1)  the  ASVAB  GT  scores’  prediction  of  Passing/7257 
Qualification,  (2)  the  MATC-  Suitability  Test  scores’  prediction  of  Passing/7257 
Qualification,  and  (3)  regression  weighted  composite  scores’  (made  up  of  ASVAB  GT 
and  MATC-ST)  of  Passing/7257  Qualification. 

The  Marines  currently  use  the  ASVAB  as  a  selection  screening  tool  to  gain  access  to 
the  ATC  School;  therefore,  GT  Score  was  the  first  variable  entered  (model  1),  followed  by 
the  MATC-ST  P  score  (model  2),  and  the  resulting  composite  (model  3).  The  same 
model  process  was  conducted  for  MATC-ST  Q  score  (model  2a)  and  the  resulting 
composite  (model  3a).  Results  of  the  regression  analysis  are  presented  in  Table  4 
(weights  are  standardized  beta  ((3)  weights). 


11 


Table  4 

ATC  Hierarchical  Regression  Validity  Analysis 


Standardized 

Measure 

Regression 

Coefficient 

R2 

AR2 

F 

AF 

t 

Model  1:  GT  Score 

.296 

.087 

— 

30.449* 

— 

5.518* 

Model  2:  MATC-ST  P 

.293 

.086 

— 

7.401* 

— 

3.070* 

Model  3:  GT  Score  & 

.296 

.149 

.085 

30.449* 

30.449 

5.518* 

MATC-ST  P 

.385 

.143 

22.759 

4.771* 

Model  2a:  MATC-ST  Q 

.416 

.173 

— 

23.266* 

— 

4.823* 

Model  3a:  GT  Score  & 

.045 

.158 

.002 

22.807* 

.222 

.471 

MATC-ST  Q 

.416 

.171 

11.536 

4.776* 

a.  Dependent  Variable:  Passing/7257  Qualification;  ^Statistically  significant  at  the  pc.OOl  probability 
level. 

The  Table  4  results  showed  that,  as  expected  from  the  results  in  table  2.2,  GT  Score 
alone  (Model  1)  predicts  Passing  Schoolhouse  performance  at  a  statistically  significant 
level  (Model  1;  P=.296,  R2=.o87,  F=30.449,  t  =5.518,  p<.ooo),  accounting  for  8.7%  of 
the  variance  in  performance.  When  the  MATC-ST  P  was  added  into  the  next  step  (Step 
2)  of  the  hierarchical  regression  analysis,  the  results  were  also  statistically  significant 
(Model  2;  P=.293,  R2=.o86,  F=7.40i,  t=3.070,  p<.ooo).  Together,  the  MATC-ST  P  and 
GT  Score  account  for  14.9%  of  the  variance  in  performance. 

A  composite  of  the  two  variables  entered  in  the  third  model  (Step  3)  improved  the 
prediction  over  MATC-ST  P  alone  (Model  3;  P=-385,  R2=.i49,  AR2  =.143,  F=30.449,  AF2 
=  22.759,  t=4-771>  p<.ooo).  The  composite  of  GT  Score  and  the  Suitability  Passing  test 
score  (R2=.i49),  suggests  that  the  predictive  validity  of  the  components  work  together 
additively  to  increase  the  validity  of  the  prediction  with  regard  to  Passing  performance. 

In  Table  4  results  showed  that,  the  MATC-ST  Q  predicts  7257  Qualification 
performance  at  a  statistically  significant  level  (Model  2a;  P=.4i6,  R2=.i73,  F=23.266, 
t=4.823,  p<.ooo),  accounting  for  17.3%  of  the  variance  in  performance.  When  the  GT 
Score  and  MATC-ST  Q  were  hierarchically  regressed  onto  7257  Qualification 
performance,  the  results  were  also  statistically  significant  (Model  3a;  P=.4i6,  R2=.is8, 
AR2  =.171,  F=22.8o7,  AF2  =11.536, 1=4.776,  p<.ooo). 

Together,  the  MATC-ST  Q  and  GT  Score  account  for  15.8%  of  the  variance  in  7257 
Qualification  performance.  The  composite  of  GT  Score  and  the  MATC-ST  Q  score 
(R2=.i58),  suggests  that  the  predictive  validity  of  the  components  work  together 
additively  to  increase  the  validity  of  the  prediction  with  regard  to  7257  Qualification 
performance. 

Figure  1  is  a  box  plot  graph  illustrating  the  recommendation  of  establishing  a 
conservative  cutscore  for  the  MATC-ST  P  that  utilizes  the  combined  GT  and  MATC- 
Suitability  Test  P  as  the  predictor  (R2=.i49).  R2  (variance  overlap)  is  the  proportion  of 
the  Passing  score  that  is  accounted  for  by  the  predictor.  The  square  root  of  this  value  is 
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the  correlation.  The  graph  also  indicates  setting  a  conservative  cutscore  will  keep  the 
lowest  passing  score  and  drop  only  failures/revocations. 

Figure  1.  Establishing  a  Conservative  Cutscore  for  MATC-ST  Passing 
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*Note:  n=321  Marine  ATCs 

Together,  GT  Score  and  MATC-  Suitability  Screening  Test  Passing  score  account  for 
14.9%  of  the  variance  in  Passing  performance;  and  GT  Score  and  MATC-  Suitability 
Screening  Test  7257  Qualification  score  account  for  15.8%  of  the  variance  in  7257 
Qualification  performance.  The  “variance  accounted  for”  metric  interpretation  of  a 
correlation  provides  meaningful  insights  into  the  value  of  a  selection  &  classification 
instrument.  That  is,  predictor  test  developers  strive  for  100%  performance  variability 
accounted  for  by  the  predictor  (a  correlation  of  1.0).  However,  to  set  a  cutscore  for  an 
operational  selection  standard,  the  correlation  coefficient  (validity  coefficient)  is 
required. 

The  MATC-ST  P  and  Q  have  been  shown  to  be  predictive  of  success  for  Marine  ATCs 
in  Passing  and  7257  Qualification.  Success  is  determined  by  passing  the  MOS 
Schoolhouse  and/or  completing  7257  qualification,  therefore  the  higher  the  MATC-ST 
score  the  higher  the  probability  to  complete  ATC  School  and  7257  qualification.  The 
expectation  is  that  the  relationship  will  hold  up  for  future  ATC  Marines  at  the 
schoolhouse  and  setting  a  conservative  cutscore  is  recommended  for  noncognitive 
MATC-ST  P  score  at  this  time.  A  near  term  goal  would  be  to  evaluate  the  positive  effects 


13 


of  this  operational  cutscore  for  cost  savings  and  increased  ATC  operational 
performance. 


Other  Topics 


Cost-Utility  Analysis 

An  additional  topic  of  interest  is  the  positive  impact  that  the  MATC-ST  had  on 
predicted  cost  savings  and  diversity.  From  the  ATC  schoolhouse  passing  sample 
(n=32i),  we  estimated  the  cost  of  failure/revocation  before  and  after  the 
implementation  of  the  MATC-ST.  Total  cost  per  ATC  Marine  was  estimated  at  $28,  671. 
This  cost  was  estimated  from  three  costs:  9  month  E-3  Salary  +  .33  benefits=  $21,391; 
Schoolhouse  cost  per  student  $2280;  and  Permanent  Change  of  Station  (PCS)  cost= 
$5000.  The  “.33  benefits”  is  an  estimated  fringe  benefit,  which  is  the  cost  that  the 
employer  pays  for  health,  dental,  life,  social  security  tax,  etc.  (This  figure  may  differ  for 
the  military  but  is  a  realistic  and  comparable  cost  estimate  for  employers.).  These 
figures  are  conservative  cost  estimates  and  do  not  take  into  account  any  implicit  costs 
such  as  facilities,  lost  wages,  overtime,  instructor  salary,  etc.  Using  the  current  selection 
measures  it  is  predicted  that  the  Marine  ATC  MOS  has  a  failure  cost  of  $3.1  million 
considering  its  current  34%  failure  percentage.  With  the  implementation  of  the  MATC- 
ST  passing  it  is  predicted  that  the  Marine  ATC  MOS  will  reduce  its  failure  percentage  to 
27.1%  resulting  in  a  savings  of  at  least  $635K  per  year.  As  a  conservative  estimate  for 
every  1%  drop  in  the  failure  percentage  the  MATC-ST  will  have  a  savings  of  over  $92K. 

Diversity  Discussion 

From  the  ATC  schoolhouse  sample  (n=405),  we  sorted  GT  Scores  from  highest  to 
lowest  and  we  selected  the  top  100  GT  scorers.  The  diversity  picture  is  illustrated  in 
Figure  2  and  annotated  here:  White  (80%)  and  Non-white  (20%);  Male  (96%)  and 
Female  (4%). 

We  then  sorted  the  GT  &  MATC-ST  P  scores  highest  to  lowest  and  we  selected  the 
top  100  scorers.  The  diversity  picture  is  illustrated  in  Figure  3  and  annotated  here: 
White  (78%)  and  Non-white  (22%);  Male  (93%)  and  Female  (7%). 
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Figure  2.  Top  100  GT  Score  and  Diversity 


Figure  3.  Top  100  GT  Score  &  MATC-ST  P  and  Diversity 


The  addition  of  the  MATC-ST  can  improve  the  ATC  screening  process  through  fair, 
valid  screening  improvements  that  not  only  predict  high  performance  in  the  operating 
forces  (Walker,  Farmer,  &  Roberts,  2012)  and  schoolhouse  (Walker,  Farmer,  &  Roberts, 
2013),  but  also  increase  diversity  within  the  Air  Traffic  Control  field. 

Recommendations 

At  the  time  of  this  report  the  Marine  ATC  Suitability  Screening  Test  (MATC-ST)  is 
shown  to  be  predictive  of  success  for  Marine  ATCs  in  the  operating  forces  (Phase  I 
results)  and  the  ATC  schoolhouse  (Phase  II  results).  The  Marine  ATC  Suitability  Test 
(MATC-ST)  score  is  a  valid  algorithm  derived  from  dimensions  of  NCAPS.  It  is 
recommended  to  set  a  conservative  cutscore  for  the  noncognitive  MATC-ST  P  score  at 
this  time  to  be  used  for  selection  prior  to  MOS  selection  for  Marine  ATCs. 
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Further  data  collection  and  analysis  from  Phase  III  cognitive  beta  testing  is 
recommended  to  provide  confirmation  of  the  additional  cognitive  test’s  predictive 
validity,  both  for  training  performance  and  ATC  job  performance.  It  is  best  practice  to 
combine  concurrent  validity  with  predictive  validity  from  the  ATC  Schoolhouse  in  order 
to  recommend  any  conservative  cutscores  and/or  implementation  into  the  Marine  Corps 
Air  Traffic  Control  selection  and  classification  process.  (This  recommendation  has  been 
submitted  as  a  Phase  III  proposal  Addendum  2  and  is  pending  at  the  time  of  this 
report). 

As  of  the  date  of  this  report  Phase  I,  Phase  II  and  Phase  III  results  were  briefed  to 
project  officers.  It  is  important  to  note  that  Phase  I  results  were  from  operating  forces 
and  Phase  II  results  were  from  the  schoolhouse.  Phase  III  results  recommended  the  best 
overall  MATC-ST  algorithm  for  predicting  Marine  ATC  performance  as  defined  by 
project  officers  (e.g.  time  to  qualify,  GPA,  behaviorally  anchored  rating  scales,  etc.). 

If  the  MATC-ST  is  to  be  utilized  for  selection/classification  it  is  recommended  that  it 
be  used  only  in  addition  to  the  ASVAB  (GT)  in  a  hurdle  approach,  and  implemented  at 
the  Military  Entrance  Processing  Station  (MEPS)  or  similar  establishment  prior  to  a 
Marine  ATC  MOS  designation/assignment.  This  will  reduce  attrition/training  costs,  and 
increase  the  quality  of  Marines  selected  for  the  ATC  MOS. 

Limitations 

The  length  of  this  study  was  approximately  one  year.  A  hindering  factor  was  allowing 
enough  time  for  performance  measures  to  be  collected  (schoolhouse  graduation  and 
7257  qualification  based  on  9  month  MARADMIN  requirement).  The  timing  did  not 
have  an  impact  on  results  other  than  delaying  Phase  III  analysis  in  order  to  allow 
enough  time  for  performance  measures  to  be  collected.  Although,  NCAPS  allowed  for 
maximum  participation  among  Marine  Air  Traffic  Controllers  in  the  schoolhouse 
because  it  was  web  enabled  and  administered  at  the  beginning  of  training;  the 
performance  data  collection  took  much  more  time  to  collect.  It  was  necessary  for  an 
adequate  amount  of  time  to  pass  to  collect  such  data  from  the  ATC  schoolhouse  and 
Operational  Forces. 

Another  limitation  at  the  time  of  this  report  is  that  the  ATC  Cognitive  prototype  data 
and  analysis  is  not  complete  for  the  Operational  Forces.  The  Cognitive  prototype  was 
developed  and  demonstrated  to  all  sponsors  and  administrators,  however  actual  beta 
testing  began  in  September  2013.  Further  analysis  will  need  to  be  performed,  as 
recommended  for  Phase  III  Addendum  2  to  include  ATC  schoolhouse  participants  and 
to  allow  for  a  larger  sample  size  and  any  range  restriction.  An  additional  report  would 
need  to  be  produced  at  the  time  of  Phase  III  Addendum  2  data  analysis  completion  for 
any  cognitive  test  implementation  recommendations. 

The  authors  tried  to  get  a  sample  that  was  most  representative  of  the  population; 
however  it  is  an  understandable  limitation  that  the  sample  may  not  be  as  representative 
as  the  population. 
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Conclusion 


This  report  further  concluded  findings  from  the  ATC  operating  forces  (Phase  I)  and 
schoolhouse  (Phase  II)  study  that  NCAPS  dimensions  calculated  specifically  for  the 
Marine  Air  Traffic  Control  community  can  predict  success  in  training  as  well  as  on  the 
job.  Phase  III  of  the  Marine  Air  Traffic  Control  Suitability  Screening  Test  (MATC-ST) 
project  focused  on  passing  performance  and  7257  Qualification  to  allow  for  a  larger 
sample  size  and  any  range  restriction,  predicting  passing  and  7257  Qualification  in  ATC 
schoolhouse  graduates  (7251).  Recommendations  were  made  for  implementation  of  an 
operational  conservative  cutscore  for  MATC-ST  Passing.  The  expectation  is  that  the 
relationship  will  hold  up  for  future  ATC  Marines,  and  that  a  near  term  goal  would  be  to 
evaluate  the  positive  effects  of  such  cutscore  on  training  cost  savings  and  operational 
performance.  Phase  III  Addendum  2  also  recommends  the  continued  validation  of  the 
MATC-ST  cognitive  prototype  addition. 

MATC-ST  implementation  can  improve  the  Marine  ATC  screening  process  through 
better  schoolhouse  performance,  better  operating  forces  performance,  and  increased 
diversity  through  fair,  valid  screening  improvements.  As  with  any  test,  it  is  important  to 
continue  to  validate  these  tools  considering  changes  in  curriculum  and  additions  of 
simulation  based  training  in  order  to  see  if  the  ASVAB  composites  and  MATC-ST 
cutscores  are  adequate  for  producing  a  job  ready  Marine. 
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Appendix  A: 

PowerPoint  Instructions  for  Assessments  1-4 


A-0 


Assessment  1  Tasks 


Assessment  1  requires 

-  memorizing  a  sequence  of  spatial  positions,  words,  or 
numbers 

while  responding  to  various  processing  tasks 


A-i 


Sequences  To  Be  Memorized 

First  you  will  see  one  of  three  types  of  sequences  (made  up  of  spatial, 
1  verbal,  or  numeric  stimuli)  that  you  must  commit  to  memory 


Spatial 


Verbal 


Nuneric 


Response  Grid 


•  After  you  have  seen  a  full  sequence,  you  will  respond  by 
entering  the  stimuli  in  the  order  in  which  they  appeared 


A-2 


Processing  Tasks 


There  are  also  three  types  of  "processing  tasks"  that 
you  must  respond  to  during  the  assessment 

Rotations 

-  Sentences 

Equations 

These  tasks  are  timed  and  you  need  to  answer  at  least 
75%  of  them  correctly  during  each  sequence 


Processing  Task  Details 
(Rotations) 


For  the  rotation  processing  task,  you  must  decide  if  the 
figure  on  the  right  represents  what  the  figure  on  the  left 
would  look  like  if  the  "frame"  on  the  left  were  rotated  to 
match  the  "frame"  in  the  middle. 

In  this  case,  the 
correct  response 
would  be  "FALSE 


Processing  Task  Details 
(Sentences  and  Equations) 


•  For  the  sentence  and  equation  processing  tasks,  you 
must  decide  if  the: 

-  sentence  word  order  is  correct  or  scrambled 

-  equation  is  true  or  false 


The  intervals  became  very  maddening  of  silence;  the  delay 


a 


Task  Order 


Now  that  you  have  seen  the  types  of  tasks,  here  is  the 
order  you  can  expect  during  the  assessment: 

-  1)  Spatial  sequences 
2)  Verbal  sequences 

-  3)  Numeric  sequences 

-  4)  Spatial  processing  (rotations) 

5)  Verbal  processing  (sentences) 

6)  Numeric  processing  (equations) 

-  7)  Sequences  with  a  processing  task  between  each  item 

Remember,  you  must  try  your  best  to  both  remember 
„  the  sequences  and  solve  the  processing  tasks  correctly 
 for  each  set 


A-5 


Assessment  2  Tasks 


Assessment  2  requires 

-  scanning  a  set  of  objects  (called  "Contacts") 
continuously 

-  while  maneuvering  another  object  over  targets 


Contact  Scanning 


•  The  left  side  of  the  screen  below  shows  a  set  of  contacts  that  will 
be  moving  around  the  screen  during  the  assessment. 

•  You  must  constantly  scan  these  contacts  to  make  sure  the  value 
shown  below  the  line  for  each  contact  falls  within  the  acceptable 
range  of  values. 

In  this  image,  you  will  see 
that  contact  "D44"  has  a 
value  (875)  that  falls 
outside  the  acceptable 
range  (360-710). 
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Contact  Reporting 

> 

•  When  you  spot  a  contact  that  is  outside  of  the  acceptable  range, 
you  will  click  the  button  "Report  Contact" 

•  The  screen  will  change  as  shown  below,  and  you  must  enter  the  ID 
number  of  the  contact 


In  this  case, 

you 

would  enter 

44  1 

(leaving  out 

the\  1 

first  letter  of 

"D44") 

Enter  ID  for  Contacts  wrtti  jfres  falling  beyond  the  range  of  (360  •  710) 

1  44 

2 
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Assessment  3  Tasks 


Assessment  3  requires 

-  directing  orange  and  green  circles  towards 
designated  goals  ("checks")  while  avoiding  any  circles 
touching  obstacles  ("X's") 

-  solving  analogies 


Initial  Arrangement 


Each  "puzzle"  in  the  assessment  will  begin  with  four  circles  (two 
green  and  two  orange)  sitting  above  the  top  of  the  8x8  grid. 

You  must  place 
the  shapes  at 
your  disposal 
on  the  grid  in 
such  a  way  that 
all  four  circles 
will  end  on  a 
goal. 

You  have  20 
minutes  to  solve 
as  many  puzzles 
as  possible. 


Adding  /Removing  Shapes 

> 

•  To  add  a  shape  to  the  grid,  click  on  the  shape  you  want  to  use, 
then  click  on  the  spot  in  the  grid  you  want  it  to  appear. 


CO 

o' 


Each  shape  will 
affect  orange 
and  green  circles 
differently. 

To  remove  a 
shape,  click  the 
"Remove  Object" 
button,  then 
click  on  the 
shape  you  want 
to  remove 


First,  click  the  shape... 


Then  click  on  the  grid... 


The  Effect  of  Shapes 


•  When  you  press  "Start,"  each  of  the  four  circles  will  start  moving  toward 
the  bottom  of  the  grid 

•  When  a  circle  encounters  a  shape,  its  direction  of  movement  will  change  to 
match  the  direction  shown  in  the  legend  on  the  right  of  the  screen. 


In  the  assessment,  a  set  of 
directional  arrows  will  be 
shown  here.  - 


These  arrows  will  indicate 
how  green  or  orange 
circles  will  react  when  they 
encounter  each  shape 
placed  on  the  grid. 


V 

X 

X 

X 

X 

y 

y 

y 

X 

B 

B 

y 
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Analogies 


•  After  20  minutes  working  on  the  puzzles,  you  will  have 
five  minutes  to  solve  a  series  of  analogies. 

•  You  will  see  statements  such  as: 

-  PENCIL  is  to  PAPER  as  BRUSH  is  to _ 


»  In  this  case  you  would  choose  the  best  of  four 
options  to  complete  the  analogy 

-  ROAD  is  to  CAR  as  WATER  is  to  JET 

»  In  this  case  you  would  choose  either  TRUE  or 
FALSE  depending  on  the  accuracy  of  the  analogy 


A-n 


Assessment  4  Tasks 


Assessment  4  requires 

-  filling  orders  for  equipment  by  running  a 
"manufacturing"  system 
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Orders 


> 

•  Orders  are  shown  on  the  bottom  right  of  the  screen.  Each  order  specifies 
an  amount  of  one  or  two  types  of  items  to  be  produced  in  a  certain 
amount  of  time. 


When  your 
inventory 
contains  enough 
of  each  type  of 
item  to  fill  an 
order,  you  can 
click  the  "Ship 
Order"  button 
next  to  the  order 
to  fill  it. 

Payment  for  the 
order  is  given 
immediately,  and 
is  reflected  in 
your  CAPITAL 
displayed  below 
the  orders. 


Manufacturing 


•  In  order  to  manufacture  items,  you  must  process  raw  materials 
through  a  series  of  machines. 


Dragging  and 
dropping  a  box 
into  a  machine 
will  produce  a 
box  with  the 
letter  of  that 
machine  on  i 


If  an  empty  box  is 
dropped  into  the  "C"  machine, 
a  "C"  bo  . 


Manufacturing  (Continued) 

> 

•  You  must  use  multiple  machines  on  the  same  box  to  produce 
finished  products. 


Machines  can 
only  process 
one  box  at  a 
time. 

The  order  in 
which  elements 
are  added  to  a 
product  is 
irrelevant. 


Each  machine  has  a  status 
indicator  that  describes  its 
state.  If  the  status  is  "FREE,"  a 
new  box  can  be  dropped  in  the 
machine. 


Dropping  a  "C"  box  into  the  "D" 
machine  would  produce  a 
completed  "CD" 

Dropping  a  "D"  box  into  the  "C" 
machine  would  also  produce  a  "CD" 


CAPITAL 

$100.00 


Inventory 


INVENTORY 


R:  1 
T:  1 
V:  1 


When  boxes  are  produced,  they  begin  moving  up  the  right 
conveyor  to  the  Warehouse.  Once  in  the  Warehouse,  they  become 
part  of  the  inventory^ 

In  order  to  ship 
products,  they 
must  be  in  the 
inventory. 

Storing  products 
in  the  Warehouse 
is  not  free, 
however.  There 
is  a  small  charge 
associated  with 
keeping  products 
in  inventory. 

CAPITAL 
-$2.60 


Capital  decreases  slowly  as  long 
as  there  are  products  in  the 
inventory. 


Reprocessing  and  Recycling 

> 

•  Products  in  the  inventory  can  be  "reprocessed"  by  clicking  on  an  item  in 
the  inventory  and  pressing  the  "Reprocess"  button. 


•  Reprocessed 
boxes  will  show 
up  on  the  left 
conveyor  and 
can  be  dropped 
into  machines 
for  further 
processing. 


Alternatively,  if 
the  box  is  no 
longer  desired, 
simply  let  it  move 
to  the  bottom  of 
the  left  conveyor 
and  it  will  be 
"Recycled"  or 
turned  into  an 
empty  box  again. 


CAPITAL 

-$2.60 


A 

Clicking  on  the  "T"  in  the  inventory, 
then  clicking  the  "Reprocess" 
button  will  send  a  "T"  box  from 
nventory  back  to  the  left  conveyor 


indicated  by  the  colored  lights 
on  the  machines.  Green 
indicates  optimal  performance; 
Yellow  indicates  suboptimal 
performance;  and  Red  indicates 
very  poor  performance. 


Machine  Maintenance 


•  The  performance  of  each  of  the  5  machines  will  degrade  over  time  if  they 
are  not  properly  maintained.  Degraded  performance  means  the  boxes  will 
take  longer  for  a  machine  to  produce. 


•  Performing  repairs 
makes  a  machine 
unavailable  for  a 
short  time,  but 
restores  it  to 
optimal  performance. 


INVENTORY 

Eiflu!  Cnmplete 

CD:  1 


Maintenance  needs  are 


CAPITAL 

$414.52 


Repairs  are  made 
by  clicking  on  the 
machine's  repair 
button. 


■rowan 


RST 


Thank  You  and  Good  Luck 


•  Click  any  key  to  enter  the  assessment 
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